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11.3.1 Introduction 

So far, we have discussed discrete-time Markov chains in which the chain jumps from the 
current state to the next state after one unit time. That is, the time that the chain spends in each 
state is a positive integer. It is equal to 1 if the state does not have a self-transition (p u = 0), or 

it is a Geometrical -p u ) random variable if p u > 0. Here, we would like to discuss 

continuous-time Markov chains where the time spent in each state is a continuous random 
variable. 

More specifically, we will consider a random process { X(t), t G [0, qo)}. Again, we assume 
that we have a countable state space 5c {0, 1, 2, •••}. IfX(0) = i, thenX(t) stays in state i for 
a random amount of time, say 7j, where 7j is a continuous random variable. At time 7j, the 

process jumps to a new state j and will spend a random amount of time T~, in that state, and so 
on. As it will be clear shortly, the random variables 7j, 7f, ••• have exponential distribution. 
The probability of going from state i to state j is shown by p : 

Similar to discrete-time Markov chains, we would like to have the Markov property, i.e., 
conditioned on the current value of X{t), the past and the future values of the process must be 
independent. We can express the Markov property as follows: for all 
0 < t, < t~ < ••• < t < t , ,, we must have 

P(X(l n+l ) =j\X(t„) = i,X(t„_,) = -,X(t,) = /,) = P(X(t n+ ,) =j\X(t„) = /). 

In particular, suppose that at time t, we know that X(t) = i. To make any prediction about the 
future, it should not matter how long the process has been in state i. Thus, the time that the 
process spends in each state must have a "memoryless" property. As it has been discussed in 
previous chapters (e.g, Chapter 41 . the exponential distribution is the only continuous 
distribution with that property. Thus, the time that a continuous-time Markov chain spends in 
state i (called the holding time) will have Exponential (A ; ) distribution, where ). i is a 

nonnegative real number. We further assume that the 2/s are bounded, i.e., there exists a real 
number M < oo such that < M, for all i G S. 

Thus, a continuous Markov chain has two components. First, we have a discrete-time Markov 
chain, called the jump chain or the the embedded Markov chain, that gives us the transition 
probabilities p Second, for each state we have a holding time parameter A ■ that controls the 

LJ l 

amount of time spent in each state. 

Note that if i is not an absorbing state, we can assume that i does not have a self-transition, i.e., 
pII = 0. The reason is that if we go from state i to state /', it's as if we never left that state. On 
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the other hand, if i is an absorbing state, we have p n = 1, and p n = 0, for all i j j. In this case, 
we have /l ■ = 0, which means that the chain will spend an infinite amount of time in the 
absorbing state i. 

Continuous-Time Markov Chains 

A continuous-time Markov chain X(t) is defined by two components: a jump chain, and a set 
of holding time parameters X- r The jump chain consists of a countable set of states 

Sc {0, 1, 2, •••} along with transition probabilities p-^. We assume p u = 0, for all non¬ 
absorbing states i E S. We assume 

1. if X(t) = i, the time until the state changes has Exponential (/L) distribution; 

2. if X(t) = i, the next state will be j with probability p-. 

The process satisfies the Markov property. That is, for all 0<t l <t 2 <-<t n <t n+l ,we 
have 


r(x(t n _ M ) =j\x« n ) - - /,) 

= P(X(t n+l )=j\X(t„) = i)- 


Let's define the transition probability Pjjif) as 

P i p) = P(X(t + s)=j\X(s) = i) 

= P(X(t) = j |X(0) = /), for all s, t E [0, oo). 

We can then define the transition matrix, P(t). Assuming the states are 1, 2, •••, r, then the state 
transition matrix for any t > 0 is given by 


'P n(0 Putt) 

C 2 |(0 PnV) 


P(t) = 


P 1.(0 1 
PlM) 


^rl(0 Pd 1 ) ■ ■ ■ Prd ) 


Let's look at an example. 
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Example 11.17 

Consider a continuous Markov chain with two states S = {0, 1}. Assume the holding time 
parameters are given by = X 1 = X > 0. That is, the time that the chain spends in each state 

before going to the other state has an Exponential^/ 1) distribution. 

a. Draw the state diagram of the embedded (jump) chain. 

b. Find the transition matrix P(t). Hint: You might want to use the following identities 


sinh(x) = --- 

e x + e“ x 
cosh(x) = --- 


00 


I 

n = 0 


x 2n+l 
(2n + 1)! ’ 


00 


I 

n = 0 


x 2n 

( 2 nV.' 


• Solution 

o a. There are two states in the chain and none of them are absorbing (since 
Xj > 0). Since we do not allow self-transitions, the jump chain must have 
the following transition matrix: 


P = 


■o 

1 


1 ' 

0 


The state transition diagram of the jump chain is shown in Figure 11.22. 


1 



Figure 11.22 - The jump chain of the continuous-time Markov 
chain in Example 11.17. 

b. This Markov chain has a simple structure. Let's find P^it). By definition 
P 00 (t ) = P(X(t ) = 0 | X(0) = 0), for all t G [0, oo). 


Assuming that X(0) = 0, X(t) will be 0 if and only if we have an even 
number of transitions in the time interval [0, t\. The time between each 
transition is an Exponential^/ 1) random variable. Thus, the transitions occur 
according to a Poisson process with parameter X. We have 


https://www.probabilitycourse.com/chapter11/11_3_1Jntroduction.php 


3/6 





10/8/2018 


Introduction 


p oo (0=n^(0 = 0|x(0) = 0) 

= P(an even number of arrivals in [0, /]) 

00 ~ 

(AO 2 ” 


77 = 0 


-At. 


, -At 


( 2 / 0 ! 

(AO 2 ” 


Jo* 2 ")! 


, -At 


e At + e At 


1 1 

= — + ~e 
2 2 


-2 At 


Next, we obtain 

^Oi(0 = ^ ~ p oo«) 

= -- -e~ nt 
2 2 

Finally, because of the symmetry in the problem, we have 

p 11 (0 = ^oo(0 

= - + - e - m , 

2 2 

^io(0 = ^oi(0 

= - - -e~ 21t 
2 2 

Thus, the transition matrix for any t > 0 is given by 


Pit) 


1 + l „-2 At 1 _ 1-2 At 

2 2 2 2 


1 - 1^-2 At 1 , 1 .- 22 / 

2 2 2 2 
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If we let t go to infinity in the above example, the transition matrix becomes 


lim P(t) = 

t —>oo 


1 1 ' 

2 2 


1 

2 


1 

2 


which means that for i = 0,1, we have 


lim P(X(t) = 0\X(0) = i ) 

t —>oo 

lim P(X(t) = 1 \X(0) = 0 

/—> oo 


1 

2 ’ 

1 

2 ' 


1 1 

In fact, tc — [~, ~] is a limiting distribution of this chain. We will discuss limiting distributions 

shortly. Before doing that let's talk about some properties of the transition matrices. First, 
similar to the discrete-time analysis, the rows of the transition matrix must sum to 1: 


Y,Pij( 0=1. for all t > 0. 

jts 

Next, we note that by definition 

P n (Q) = P(X(Q) = i\X(Q) = i) 

= 1, for all i. 

p ij(°) = P(X(0) = 71 ^( 0 ) = 0 
= 0, for all i ± j. 

We conclude that P( 0) is equal to the identity matrix, P( 0) = I. Finally, we can obtain the 
Chapman-Kolmogorov equation by applying the law of total probability and by using the 
Markov property: 

P ij (s + t) = P(X(s + t)=j\X(0) = i) 

= X P(X(s) = k\X(0) = i)P(X(s + t ) =j\X(s) = k) 
kes 

= Z p d s ) p ki( 0. for a11 t> 0. 

kes 
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The above equation can be written in the matrix fonn as follows 

P(s + t) = P(s)P(t), for all s, t > 0. 
Transition Matrix 


For a continuous-time Markov chain, we define the transition matrix/ 3 !/). The (z',y')th entry of 
the transition matrix is given by 


P lJ (t) = P(X(t)=j\X(()) = i). 
The transition matrix satisfies the following properties: 

1. P( 0) is equal to the identity matrix, P( 0) = /; 

2. the rows of the transition matrix must sum to 1, 

YjPi/O = ’’ for all t > 0; 
j£S 


3. for all s, t > 0, we have 


P{s + 0 = P(s)P(t). 


<r— previous 
next —> 
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